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GAO 

Accountability  *  Integrity  *  Reliability 


United  States  General  Accounting  Office 
Washington,  D.C.  20548 


Accounting  and  Information 
Management  Division 


B-282546 

November  15,  1999 

The  Honorable  Jeriy  Lewis 
Chairman,  Subcommittee  on  Defense 
Committee  on  Appropriations 
House  of  Representatives 

Dear  Mr.  Chairman: 

Complete  and  thorough  end-to-end  testing  is  essential  to  provide 
reasonable  assurance  that  new  or  modified  systems  used  to  collectively 
support  a  core  business  function  or  mission  operation  will  not  jeopardize 
an  organization’s  ability  to  deliver  products  and  services  as  a  result  of  the 
Year  2000  (Y2K)  computing  problem.  This  is  especially  true  for  the 
Department  of  Defense  (DOD)  because  it  relies  on  a  complex  and  broad 
array  of  interconnected  computer  systems — including  weapons,  command 
and  control,  satellite,  inventory  management,  transportation  management, 
health,  financial,  personnel  and  payment  systems — to  carryout  its  military 
operations  and  supporting  business  functions. 

At  your  request,  we  reviewed  DOD’s  management  of  various  Year  2000- 
related  end-to-end  testing  activities.  As  part  of  our  efforts,  we  assessed  the 
U.S.  Space  Command’s  management  of  its  end-to-end  test  of  space  control 
systems  essential  to  major  theater  war  (one  of  16  operational  evaluations 
for  the  command)  and  determined  what  the  results  of  this  test  show  with 
respect  to  operational  risks  and  readiness. 1  We  briefed  Space  Command 
officials  on  our  findings  on  October  1, 1999,  and  made  a  recommendation 
to  correct  the  management  weaknesses  that  we  found.  Space  Command 
immediately  acted  to  address  our  recommendation.  We  then  briefed  your 
office  on  our  findings  and  Space  Command’s  actions  to  address  our 
recommendation  on  November  1,  1999.  The  purpose  of  this  letter  is  to 
summarize  our  briefing  to  your  office.  The  briefing  slides  that  we  presented 
to  your  office  are  in  appendix  I,  and  the  objectives,  scope,  and 
methodology  of  our  review  are  detailed  in  appendix  II.  Space  Command 
provided  oral  comments  on  our  briefing  slides,  and  we  have  incorporated 
them  as  appropriate.  We  performed  our  audit  work  from  March  through 


'DOD  refers  to  its  combatant  commands’  end-to-end  tests  as  operational  evaluations. 
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October  1999  in  accordance  with  generally  accepted  government  auditing 
standards. 


Results  ill  Brief  Year  2000  end-to-end  testing  is  an  essential  component  of  an  effective  Year 

2000  testing  program  since  Y2K-related  problems  can  affect  so  many  of  the 
systems  owned  and  operated  by  an  entity  as  well  as  systems  belonging  to 
business  partners  and  infrastructure  providers.  Moreover,  to  be  effective, 
end-to-end  testing  should  be  approached  in  a  structured  and  disciplined 
fashion.  Both  the  Joint  Chiefs  of  Staff  (JCS)  guidance  to  its  combatant 
commands  on  managing  Year  2000  operational  evaluations,2  (the  term  JCS 
uses  to  refer  to  Year  2000  end-to-end  testing)  and  our  Year  2000  test 
guidance3  define  a  number  of  key  management  controls  to  employ  when 
planning,  executing,  analyzing,  and  reporting  on  such  test  and  evaluation 
events. 

We  found  that  Space  Command’s  space  control  operational  evaluation 
satisfied  16  of  21  of  the  key  processes  prescribed  by  JCS  guidance.  For 
example,  the  Command  established  a  Y2K  task  force  to  guide  the 
evaluation  effort,  which  included  satellite/system  specialists,  test  and 
evaluation  experts,  system  analysts,  military  component  and  service 
representatives,  and  public  affairs  representatives.  Further,  the  Command 
performed  a  rehearsal  before  conducting  the  evaluation  to  ensure  that  all 
critical  systems  and  interfaces  were  operating  correctly  and  that  all  staff 
knew  their  roles  and  responsibilities. 

In  response  to  our  concerns,  Space  Command  has  taken  positive  actions  to 
address  the  remaining  five  key  processes.  Three  of  the  key  processes  were 
addressed  during  the  course  of  our  review  and  two  were  addressed  in 
response  to  a  recommendation  we  made  at  our  briefing.  During  the  course 
of  our  review,  Space  Command  began  ensuring  that  contingency  plans 
were  in  place  for  its  mission-critical  systems,  which  it  had  not  done  before 
conducting  the  space  control  operational  evaluation.  Also,  after  we  found 
that  configuration  management  procedures  were  not  always  followed 


2 Joint  Staff  Year  2000  Operational  Evaluation  Guide,  Version  3.0,  April  1, 1999. 

3 Year  2000  Computing  Crisis:  A  Testing  Guide  (GAO/AIMD-10.1.21,  issued  as  an  exposure 
draft  in  June  1998;  issued  in  final  in  November  1998). 
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while  executing  the  evaluation,4  Space  Command  initiated  an  effort  to 
ensure  that  such  procedures  are  followed  in  future  evaluations.  In  addition, 
during  our  review,  the  Command  amended  its  report  to  discuss  its  decision 
to  exclude  six  communications  systems  from  the  evaluation  and  whether 
this  adversely  impacted  the  ability  to  draw  conclusions  about  mission 
readiness. 

At  the  time  of  our  October  1, 1999,  briefing.  Space  Command  still  needed  to  • 
address  two  partially  satisfied  key  processes,  which  included  (1)  not 
documenting  whether  test  cases  for  most  intelligence  systems  met 
performance  exit  criteria  and  (2)  not  ensuring  that  1  of  29  systems  included 
in  the  evaluation  was  Y2K  compliant.  We  therefore  recommended  that 
Space  Command  amend  its  final  report  to  JCS  to  recognize  the 
uncertainties  and  risks  associated  with  its  failure  to  take  these  steps  and 
the  actions  underway  or  planned  to  address  these  uncertainties  and  risks. 
Without  taking  these  steps,  Space  Command  could  not  adequately  know 
the  Year  2000  readiness  of  critical  tasks— collecting  surveillance  and 
intelligence  data  to  disseminate  warning  messages — associated  with 
conducting  the  space  control  mission.  Because  Space  Command  has 
subsequently  amended  its  final  report  and  plans  to  ensure  that  these 
weaknesses  are  not  repeated  in  a  November  operational  evaluation  of  its 
intelligence  mission,  we  are  not  making  further  recommendations  at  this 
time. 


Background 


Space  Command’s  mission  is  to  provide  direct  support  to  combatant 
commanders  and  military  forces  through  the  use  of  space-based  satellites 
and  other  technologies  needed  for  navigation,  surveillance  and 
reconnaissance,  communications,  environmental  and  attack  warnings 
during  war  and  peacetime  operations.  To  perform  this  mission,  Space 
Command  relies  on  a  wide  array  of  information  technology  systems, 
including  command  and  control  systems,  geographically  dispersed  radar 
sites,  satellites,  communications  networks,  and  intelligence  systems. 

In  August  1998,  the  Secretary  of  Defense  directed  JCS  to  require  its 
combatant  commands,  including  Space  Command,  to  plan,  execute, 


Configuration  management  involves  establishing  product  baselines  and  systematically 
controlling  changes  made  to  those  baselines.  Without  an  effective  configuration 
management  process,  organizations  can  lose  control  of  the  software  product,  potentially 
produce  and  use  inconsistent  product  versions,  and  create  operational  problems. 
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analyze,  and  report  on  a  series  of  simulated  Year  2000  operational 
evaluations.  The  evaluations,  which  were  to  assess  whether  DOD  can 
continue  to  perform  critical  military  operations  in  a  Year  2000  environment, 
are  one  of  three  DOD  end-to-end  testing  efforts.5 

The  purpose  of  end-to-end  testing  is  to  verify  that  a  defined  set  of 
interrelated  systems,  which  collectively  support  an  organizational  core 
business  area  or  function,  interoperate  as  intended  in  an  operational 
environment  (either  actual  or  simulated) .  These  interrelated  systems 
include  not  only  those  owned  and  managed  by  an  organization,  but  also  the 
external  systems  with  which  they  interface  or  that  otherwise  support  the 
business  area  or  function.  The  combatant  commands’  core  business  areas 
or  functions  are  referred  to  as  “thin  lines.” 

The  boundaries  for  end-to-end  tests  can  vary  depending  on  a  given  business 
function’s  system  dependencies  and  criticality  to  the  organizational 
mission.  Therefore,  in  managing  end-to-end  test  activities,  it  is  important  to 
analyze  the  interrelationships  among  core  business  functions  and  their 
supporting  systems  and  the  mission  impact  and  risk  of  date-induced  system 
failures  and  to  use  these  analyses  to  define  test  boundaries.  It  is  also 
important  to  work  early  and  continually  with  functional  partners  to  ensure 
that  related  end-to-end  test  activities  are  effectively  coordinated  and 
integrated.  Table  1  summarizes  key  processes  recommended  by  JCS’Year 
2000  operational  evaluation  guidance,  which  is  consistent  with  our  Year 
2000  test  guide. 


5In  addition  to  conducting  operational  evaluations,  the  military  services  are  conducting 
system  integration  testing,  and  the  functional  business  areas,  such  as  personnel  and  health 
affairs,  are  conducting  functional  end-to-end  tests.  Each  of  these  end-to-end  testing 
activities  is  discussed  in  detail  in  Defense  Computers:  Management  Controls  Are  Critical  to 
Effective  Year  2000  Testing  (GAO/AIMD-99-172,  June  30,  1999). 
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Table  1:  Summary  of  JCS  Year  2000  Operational  Evaluation  Criteria 


Planning 

•  Specify  test  assumptions  and  limitations 

•  Establish  a  Year  2000  task  force 

•  Identify  critical  missions/tasks/systems 

•  Verify  that  systems  essential  to  mission  are  Year  2000  compliant 

•  Develop  an  operational  evaluation  plan  to  guide  event  planning  and 
execution 

•  Identify  and  schedule  support  from  other  commands,  DOD  components, 
etc. 

•  Determine  relevant  and  necessary  resources  (e.g.,  funding,  personnel, 
equipment,  etc.) 

•  Ensure  approved  Year  2000  contingency  plans  are  prepared 

•  Develop  a  risk  management  plan 

•  Identify  simulation  needs  and  establish  supporting  testing  environment 

•  Develop  data  collection  and  analysis  plan  or  approaches 

Execution 

•  Conduct  operational  evaluation  rehearsal 

•  Follow  configuration  management  policy 

•  Perform  baseline  test  for  operational  evaluation 

•  Execute  required  Year  2000  date  rollover  tests 

•  Collect  and  archive  al  Year  2000-relevant  data  and  ensure  that  systems 
are  reset  to  current  day  operations 

Analysis 

•  Categorize,  document,  and  report  Year  2000  failures 

•  Determine  mission  impact  of  Year  2000  failures 

•  Ensure  exit  criteria  are  met 

Reporting 

•  Prepare  Year  2000  reports  describing  mission  impact  and  readiness 

•  Provide  reports  to  JCS  within  required  timeframes 

Space  Command  has  already  completed  16  operational  evaluations  to 
assess  its  ability  to  manage  and  provide  combatant  support  during  a  major 
theater  war.  These  evaluations  covered  seven  mission  areas,  including 

(1)  integrated  tactical  warning  and  attack  assessment,  (2)  space  control, 
(3)  force  enhancement,  (4)  weather  support,  (5)  command  and  control  of 
space  forces,  (6)  space  operations  support,  and  (7)  space  lift.  The  space 
control  mission  area  provides  (1)  surveillance  support  to  monitor,  track, 
identify,  and  catalog  all  orbiting  space  objects  for  collision  avoidance  and 

(2)  protection  support  to  monitor,  detect,  assess,  characterize,  track,  and 
issue  warnings  about  threats,  both  natural  and  man-made,  against  United 
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States  and  allied  space  systems.  The  space  control  evaluation  was 
executed  between  March  11  and  March  25, 1999. 


Space  Command 
Implemented  Most 
Important  Management 
Processes  During  Its 
Space  Control 
Evaluation 


As  noted  in  table  2  below,  we  found  that,  for  its  space  control  operational 
evaluation,  Space  Command  satisfied  the  majority  of  the  management 
process  controls  (16  of  21)  specified  in  JCS’  operational  evaluation 
guidance. 


Table  2:  Summary  of  Space  Command’s  Satisfaction  of  JCS  Evaluation  Criteria  for 
the  Space  Control  Evaluation 


Phases 

Number  of  primary 
evaluation  criteria 

Number  of  primary 
criteria  satisfied 

Planning 

11 

9 

Execution 

5 

4 

Analysis 

3 

2 

Reporting 

2 

1 

Total 

21 

16 

Consistent  with  JCS  guidance  governing  operational  evaluation  planning, 
Space  Command  established  a  Year  2000  task  force,  which  included 
satellite/system  specialists,  test  and  evaluation  experts,  system  analysts, 
military  component  and  service  representatives,  and  public  affairs 
representatives.  It  identified  35  critical  tasks  that  it  needed  to  carry  out  the 
space  control  mission  in  support  of  a  major  theater  war.  Space  Command 
also  issued  a  directive  to  ensure  testing  resources  would  be  made  available 
for  operational  evaluations  and  earmarked  about  $8  million  for  operational 
evaluation  activities— including  the  space  control  evaluation.  Further, 
Space  Command  developed  a  test  plan  that  documented  participant  roles 
and  responsibilities,  critical  missions  and  tasks,  test  cases,  and  reporting 
requirements. 

Space  Command  also  took  effective  steps  in  executing,  analyzing,  and 
reporting  on  its  evaluation.  For  instance,  before  executing  the  operational 
evaluation,  Space  Command  performed  a  rehearsal  to  ensure  that  all 
critical  systems  and  interfaces  were  operating  correctly  and  that  all  staff 
knew  their  roles  and  responsibilities.  Before  resetting  systems  to  current 
day  operations,  Space  Command  ensured  that  thin  line  systems  were 
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assessed,  master  scenario  events  were  performed  and  deviations  were 
identified,  and  that  all  data  needed  to  make  an  assessment  of  the 
command’s  ability  to  perform  the  space  control  mission  were  collected  and 
archived. 


Space  Command  Acted 
to  Address  Three 
Partially  Satisfied  Key 
Processes 


Following  its  operational  evaluation,  Space  Command  took  action  to 
resolve  three  partially  satisfied  key  processes.  In  doing  so,  it  increased  its 
assurance  with  respect  to  the  Y2K  readiness  of  space  control  critical  tasks 
involving  intelligence  and  communications  systems. 

First,  before  conducting  its  test,  Space  Command  did  not  verify  that 
contingency  plans  were  in  place  for  the  29  systems  included  in  the 
evaluation.  Instead,  Space  Command  relied  exclusively  on  system  owners 
to  do  so.  As  noted  in  JCS  testing  guidance,  contingency  plans  identify 
alternative  systems  or  workaround  procedures  to  use  when  performing  a 
mission  in  the  event  of  a  system  disruption.  As  such,  JCS  guidance  states 
that  it  is  essential  that  commands  ensure  that  these  plans  are  in  place  prior 
to  executing  the  operational  evaluation  so  that  they  can  be  invoked  in  the 
case  of  system  failure.  Subsequent  to  the  evaluation,  Space  Command 
began  verifying  that  contingency  plans  are  in  place  for  its  mission-critical 
systems. 


Second,  while  executing  the  evaluation,  Space  Command  did  not  follow 
configuration  management  procedures.  JCS  guidance  specifies  that  system 
configurations  not  be  changed  during  testing  unless  authorized  by  the  test 
director.  During  the  space  control  evaluation,  changes  were  made  to  one 
system  after  the  baseline  for  the  evaluation  was  established  and  without 
authorization  from  the  test  director.  These  changes  contributed  to  a  “hard” 
failure  during  testing.  (Information  on  the  nature  of  the  system  failure  is 
classified).6  After  the  evaluation,  Space  Command  directed  the  17thTest 
Squadron  and  intelligence  unit  to  review  this  deviation  and  its  impact  on 
the  command’s  ability  to  determine  mission  readiness.  On  September  30, 
1999,  the  intelligence  unit  and  17th  Test  Squadron  reported  that  the 
deviation  did  not  materially  affect  mission  readiness.  To  prevent  similar 
problems  in  future  evaluations,  Space  Command  directed  the  17th  Test 
Squadron  and  intelligence  unit  to  develop  ways  to  improve  testing 


6  A  “hard”  failure  is  a  Y2K-related  failure  that  results  in  an  obvious  adverse  impact  to  the 
system.  For  example,  the  system  shuts  down,  displays  erroneous  data,  or  performs  other 
unexpected  actions. 


Page  9 


GAO/AIMD-00-30  Space  Command  Y2K  Operational  Testing 


documentation  and  procedures  with  a  special  focus  on  ensuring  that 
documentation  standards,  configuration  management  procedures,  and 
baseline  test  requirements  are  followed. 

Third,  in  reporting  on  the  evaluation,  Space  Command  did  not  specify  how 
its  exclusion  of  six  communications  systems  from  the  test  impacted  its 
ability  to  draw  conclusions  about  mission  readiness.  When  planning  the 
evaluation,  Space  Command  concluded  that  it  would  not  include  six 
communications  systems  in  the  evaluation  due  to  resource  constraints  or 
because  the  systems  were  to  be  included  in  a  future  evaluation.  As  a  result, 
Space  Command  assumed  that  communications  systems  would  be 
available  to  perform  critical  tasks  and  disseminate  time-sensitive  warnings 
to  combatant  commanders.  While  Space  Command  communicated  this 
assumption  to  JCS  in  its  operational  evaluation  plan,  it  did  not  report  on 
how  this  scope  limitation  could  adversely  affect  its  ability  to  draw 
conclusions  about  mission  readiness.  Instead,  Space  Command  reported  to 
JCS  that  critical  space  control  tasks  could  be  performed  across  the 
calendar  and  leap  year  dates  with  no  significant  impact  on  its  mission 
readiness.  Space  Command  has  since  ensured  that  omitted 
communications  systems  were  included  in  other  Year  2000  end-to-end 
testing  or  operational  evaluation  events  and  disclosed  this  limitation  in  its 
final  report  on  the  evaluation. 


Space  Command  Is 
Acting  to  Address 
Recommendation 
Made  at  the  Briefing 


At  the  time  of  our  October  1, 1999,  briefing.  Space  Command  had  not  yet 
addressed  two  partially  satisfied  key  processes.  First,  in  planning  the 
evaluation,  Space  Command  did  not  ensure  that  one  intelligence  system  to 
be  tested  was  certified  as  compliant.  Rather,  it  only  verified  that  the 
software  application  relevant  to  the  evaluation  was  compliant.  Year  2000 
compliance  of  an  application  in  isolation  is  of  very  limited  value  unless  the 
system  platform  that  it  runs  on,  as  well  as  other  applications  operating  on 
the  system,  is  compliant.  As  such,  both  JCS  guidance  and  GAO’s  end-to-end 
test  guidance  define  system,  not  application,  compliance  as  a  precondition 
to  end-to-end  testing. 

Second,  Space  Command  did  not  document  whether  intelligence  systems 
met  system  performance  exit  criteria  for  all  test  cases.  Specifically,  the 
command  was  supposed  to  show  whether  it  could  process  a  predetermined 
number  of  transactions  within  specific  time  constraints.  While  command 
officials  contend  that  this  was  done,  only  one-fifth  of  the  transactions  for 
intelligence  critical  tasks  were  documented.  Space  Command  officials 
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stated  that  it  was  too  time-consuming  for  operators  to  print  screens  for 
these  tasks  during  the  evaluation. 

At  our  briefing,  we  recommended  that  Space  Command  amend  its  final 
report  to  JCS  to  recognize  the  (1)  uncertainties  and  risks  associated  with 
its  failure  to  fully  satisfy  these  criteria  and  (2)  the  actions  it  had  underway 
or  planned  to  address  these  uncertainties  and  risks.  Space  Command 
agreed  with  this  recommendation.  It  plans  to  amend  its  final  report  to 
disclose  these  limitations  and  to  pursue  an  alternative  data  collection 
strategy  for  its  planned  November  1999  operational  evaluation  of  its 
intelligence  mission  in  order  to  verify  that  intelligence  systems/tasks  fully 
meet  performance  criteria. 


Conclusion 


By  acting  swiftly  to  address  our  recommendation,  made  during  the 
October  1, 1999,  briefing,  Space  Command  has  demonstrated  its 
commitment  to  improving  management  controls  over  Year  2000  testing 
activities  and  the  effectiveness  and  value  of  its  operational  evaluation  as 
well  as  mitigated  the  risks  associated  with  being  able  to  operate  effectively 
in  the  Year  2000.  Further,  it  has  ensured  that  DOD  managers  have  complete 
and  reliable  information  to  use  in  making  informed  military  decisions.  As  a 
result,  Space  Command  has  satisfied  the  intent  of  our  recommendation, 
and  we  are  not  making  any  further  recommendations  at  this  time. 


We  are  sending  copies  of  this  report  to  Representative  Jchn  P.  Murtha, 
Ranking  Minority  Member,  Subcommittee  on  Defense,  House 
Appropriations  Committee;  Senator  John  Warner,  Chairman,  and  Senator 
Carl  Levin,  Ranking  Minority  Member,  Senate  Committee  on  Armed 
Services;  Senator  Ted  Stevens,  Chairman,  and  Senator  Daniel  Inouye, 
Ranking  Minority  Member,  Subcommittee  on  Defense,  Senate  Committee 
on  Appropriations;  and  Representative  Floyd  Spence,  Chairman,  and  Ike 
Skelton,  Ranking  Minority  Member,  House  Committee  on  Armed  Services. 

We  are  also  sending  copies  to  the  Honorable  John  Koskinen,  Chair  of  the 
President’s  Year  2000  Conversion  Council;  the  Honorable  William  Cohen, 
Secretary  of  Defense;  the  Honorable  John  Hamre,  Deputy  Secretary  of 
Defense;  General  Henry  Shelton,  Chairman  of  the  Joint  Chiefs  of  Staff; 
Arthur  Money,  Assistant  Secretary  of  Defense  for  Command,  Control, 
Communications,  and  Intelligence;  and  the  Honorable  Jacob  Lew,  Director, 
Office  of  Management  and  Budget.  Copies  will  also  be  made  available  to 
others  upon  request. 
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Should  you  or  your  staff  have  any  questions  concerning  this  report,  please 
contact  me  at  (202)  512-6240. 1  can  also  be  reached  by  e-mail  at 
brockj.aimd@gao.gov.  Other  points  of  contact  and  key  contributors  to  this 
report  are  listed  in  appendix  III. 

Sincerely  yours, 


Jack  L.  Brock,  Jr. 

Director,  Governmentwide  and  Defense 
Information  Systems 
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Briefing  on  Results  of  GAO  Review  of 
SPACECOM  Space  Control  Y2K  OPEVAL 


Results  of  GAO  Review  of 
SPACECOM  Space  Control  Y2K  OPEVAL 

House  Appropriations  Committee 

November  1, 1999 


Page  14 


GAO/AIMD-00-30  Space  Command  Y2K  Operational  Testing 


Appendix  I 

Briefing  on  Results  of  GAO  Review  of 
SPACECOM  Space  Control  Y2K  OPEVAL 


i 

GAO 

Accountability  *  Integrity  *  Reliability 


Introduction 


•  In  August  1998,  the  Secretary  of  Defense  directed  the  Commanders- 
in-Chief  (CINC),  who  are  responsible  for  Defense’s  unified  combatant 
commands,  to  plan,  execute,  and  report  on  a  series  of  simulated  Year 
2000  operational  evaluations  (Y2K  OPEVALs). 

•  The  CINC  Y2K  OPEVALs  are  one  of  three  Defense  Y2K  end-to-end 
test  and  evaluation  efforts.  GAO’s  Y2K  Test  Guide  advocates  end-to- 
end  testing,  which  is  testing  performed  to  verify  that  a  defined  set  of 
interrelated  systems  (i.e.,  systems  that  collectively  support  an 
organizational  core  business  function  or  operation)  interoperate  as 
intended  in  a  Y2K  environment. 

The  CINC  core  business  functions/operations  are  referred  to  as  “thin 
lines.”  The  “thin  lines”  consist  of  critical  tasks,  as  well  as  systems  that 
perform  critical  tasks. 
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Briefing  on  Results  of  GAO  Review  of 
SPACECOM  Space  Control  Y2K  OPEVAL 


•  At  the  request  of  the  Chairman,  House  Appropriations  Committee, 
Defense  Subcommittee,  GAO  is  reviewing  selected  OPEVALs  to 
determine: 

(1)  if  the  OPEVAL  was  planned,  executed,  and  documented  in 
accordance  with  DOD  guidelines,  and 

(2)  what  the  OPEVAL  results  indicated  concerning  readiness  and  risks. 

•  The  OPEVALS  reviewed  by  GAO  included  those  conducted  by 
Space  and  Transportation  Commands  and  were  selected  in 
collaboration  with  the  Defense  Inspector  General  (IG)  to  ensure: 

-  appropriate  coverage  of  all  CINC  OPEVALs,  and 

-  no  duplication  of  effort. 
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SPACECOM  Space  Control  Y2K  OPEVAL 


i 

GAO 

Accountability  *  Integrity  *  Reliability 


Scope  and  Methodology 


This  briefing  addresses  the  Space  Command  OPEVAL  for  Space 
Control.  To  satisfy  objective  (1).  we 


•  reviewed  the  OPEVAL  plan,  testing  documents/records,  and  test 
results/reports; 

•  interviewed  Space  Command  officials  responsible  for  Y2K 
OPEVAL  planning,  execution,  and  reporting  tasks;  and 

•  compared  Space  Command’s  planning,  execution,  analysis,  and 
reporting  actions  with  Defense  OPEVAL  guidance. 
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i 

GAO 

^ ^^^^^Accountability  *  Integrity  *  Reliability 


Scope  and  Methodology 


•  To  satisfy  objective  (2),  we 


•  reviewed  Space  Command’s  OPEVAL  results,  7-and  30-day 
reports,  and  system  problem  tracking  reports  and 

•  interviewed  Space  Command  officials  and  analysts  responsible  for 
developing  OPEVAL  assessment  methodologies,  interpreting 
evaluation  metrics,  and  ensuring  that  evaluation  exit  criteria  were 
met. 


•  On  October  1 ,  1999,  we  briefed  Space  Command  leadership  on  the 
results  of  our  review  and  made  a  recommendation  to  address  our 
findings.  In  agreeing  to  our  recommendation,  Space  Command  has 
taken  action  to  address  weaknesses  identified  during  the  review  and 
plans  to  amend  its  final  OPEVAL  report  accordingly. 

•  We  performed  our  work  from  March  1 999  through  October  1 999  in 
accordance  with  generally  accepted  government  auditing  standards. 
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Briefing  on  Results  of  GAO  Review  of 
SPACECOM  Space  Control  Y2K  OPEVAL 


i 

i ^  GAO 

Accountability  *  Integrity  *  Reliability 


Background 


POD  OPEVAL  Overview 

•  To  assist  the  CINCs  in  planning,  documenting,  executing, 
analyzing,  and  reporting  OPEVALs,  the  Joint  Staff  issued  OPEVAL 
guidance.  The  guidance  is  divided  into  phases: 

-  planning, 

-  execution, 

-  analysis,  and 

-  reporting 

•  The  OPEVAL  guidance  is  consistent  with  GAO’s  end-to-end  testing 
guidance  and  DOD’s  Y2K  management  plan. 
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Briefing  on  Results  of  GAO  Review  of 
SPACECOM  Space  Control  Y2K  OPEVAL 


i 

£GAO 

Accountability  *  Integrity  *  Reliability 


Background 


Space  Command  OPEVAL  Events’  Status 


•  Space  Command  is  responsible  for  providing  continuous  and  real-time 

(1)  warnings  of  air  or  space  attacks  against  North  America  and  (2)  space 
control,  surveillance,  communications,  and  intelligence  support  to  military 
operations  worldwide. 


•  To  fulfill  its  missions,  Space  Command  depends  heavily  on  information 
systems  and  technology,  including  satellites,  geographically  dispersed 
radars/sensors,  ground  relay  terminals/stations,  and  communication 
networks. 
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i 

GAO 

Accountability  »  integrity  *  Reliability 


•  Space  Command  identified  7  “thin  lines”  (missions)  to  be 
operationally  evaluated:  (1)  Integrated  Tactical  Warning  and  Attack 
Assessment,  (2)  Space  Control,  (3)  Force  Enhancement, 

(4)  Weather  Support,  (5)  Command  and  Control  (C2)  of  Space 
Forces,  (6)  Space  Operations  Support,  and  (7)  Space  Lift. 

•  For  the  7  “thin  lines”,  Space  Command  identified  92  critical  tasks1 
and  86  supporting  systems. 

•  The  following  table  describes  the  status  of  Space  Command’s  16 
OPEVALs  and  the  Chairman’s  Contingency  Assessment  (CCA). 


1  The  number  does  not  include  critical  tasks  for  the  first  two  OPEVALS  because  critical  tasks  were  not 
identified  in  reports  for  the  first  two  OPEVALS. 

-  ■  ~ 


Background 
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GAO 

Accountability  *  Integrity  *  Reliability 


Background 


Thin  Line" 


OPEVAL  -  North  Integrated  Tactical 
American  Warning  and  Attack 

Aerospace  Defense  Assessment 
Command  (ITWAA) 

(NORAD) _ 

OPEVAL  -  NORAD  ITWAA 


Evaluation  Events 


Schedule 


December  2-4, 
1998 


Reported  mission  ready 
0  "soft”1  failures 
0  "hard" 2  failures 


OPEVAL 


ITWAA 


Space  Control 


February  16-28,  Reported  mission  ready 
1999  2  "soft"  failures 

_ 1  “hard"  failure _ 

March  15-25,  Reported  mission  ready 
1999  2  “soft"1  failures 


3  "hard  failures 


OPEVAL  -  Central  Command  and  April  6-12,1 999 

Command  Control  (C2)  - 

Space  Forces 


April  6-12,  1999  Reported  mission  ready 
1  "soft"1  failure 
0  "hard"2  failures 


'A  "soft"  failure  is  a  Y2K-related  failure  that  is  not  immediately  discernable.  The  effect  may  be  cumulative 
and  require  several  hours,  days,  or  longer  to  manifest  itself. 

2 A  “hard”  failure  is  a  Y2K-related  failure  that  results  in  an  obvious  adverse  impact  to  the  system.  For 
example,  the  system  shuts  down,  erroneous  data  is  displayed,  or  unexpected  actions  occur. 

3  Information  classified  by  the  Department  of  Defense. 
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i 

£GAO 

Accountability  *  Integrity  *  Reliability 


Background 


Evaluation  Events 

“Thin  Line” 

Schedule 

Results 

OPEVAL  -  Central 
Command 

Global  Positioning 
System  -  Space 
Support/Space 

Force  Enhancement 

April  23-May  1 , 
1999 

Reported  mission  ready 

0  “soft”  failures 

0  “hard"  failures 

OPEVAL  -  Central 
Command 

Theater  Ballistic 
Missile  Warning 
(TBMW)  -  Space 
Support/Space 

Force  Enhancement 

April  22-30, 

1999 

Reported  mission  ready 

0  “soft”  failures 

1  “hard”  failure 

OPEVAL  -  Central 
Command 

Space  Support/ 
Satellite  Control 

May  1-2  & 

May  10-11, 

1999 

Reported  mission  ready 

3  “soft”1  failures 

0  “hard”2  failures 

OPEVAL -Central 
Command 

Terrestrial  Weather 
-  Space  Support/ 
Space  Force 
Enhancement 

May  10-17, 

1999 

Reported  mission  ready 

1  “soft”1  failure 

0  “hard”2  failures 
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l 

fflAO 

Accountability  *  Integrity  *  Reliability 


Background 


Evaluation  Events 

“Thin  Line” 

Schedule 

“ 

Results 

OPEVAL 

Communications  - 
Space  Operations 
Support 

June  9-15,  1999 

Reported  mission  ready  \ 

0  “soft”1  failures 

0  “hard”2  failures 

CCA4 

Intelligence, 
Surveillance,  & 
Reconnaissance 

June  14-18,1999 

Classified3 

OPEVAL 

Space  Weather 

June  19-July  14, 1999 

Reported  mission  ready 

0  “soft”1  failures 

0  “hard”2  failures 

OPEVAL - 
Central  Command 

TBMW  -  C2  Space 
Forces 

July  15-31, 1999 

Reported  mission  ready 

0  “soft”  failures 

1  “hard”  failure 

OPEVAL  -  Central 
Command 

Communications  - 
Space  Force 
Enhancement 

July  27, 1999 

Reported  mission  ready 

1  “soft”  failure 

0  “hard  failures 

'The  CCA  was  designed  to  evaluate  the  ability  of  unified  commands  to  perform  missions  in  an 
environment  degraded  by  Y2K  failures. 
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i 

GAO 

Accountability  *  Integrity  *  Reliability 


Background 


Evaluation  Events 

"Thin  Line” 

Schedule 

Results 

OPEVAL 

Space  Lift 

July  22, 1999 

Reported  mission  ready 

0  “soft”  failures 

0  “hard”  failures 

OPEVAL 

Space  Lift 

July  27, 1999 

Reported  mission  ready 

0  “soft”  failures 

0  “hard”  failures 

OPEVAL 

Space  Lift 

August  2-6,  1999 

Reported  mission  ready 

0  “soft”  failures 

0  “hard”  failures 

OPEVAL 

Space  Lift 

September  1-3,  1999 

Reported  mission  ready 

0  “soft”  failures 

0  “hard”  failures 
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i 

£6A0 

Accountability  *  integrity  *  Reliability 


Space  Control  OPEVAL 

•  The  Space  Control  “thin  line”  includes  providing  (1)  surveillance 
support  to  monitor,  track,  identify,  and  catalog  all  orbiting  space 
objects  for  collision  avoidance  and  (2)  protection  support  to  monitor, 
detect,  assess,  characterize,  track,  and  issue  warnings  about  threats, 
both  natural  and  man-made,  against  U.S.  and  allied  space  systems. 

•  The  Space  Control  OPEVAL  was  conducted  in  collaboration  with 
other  DOD  organizations,  including  Air  Force,  Army,  and  Navy  Space 
Commands.  It  was  intended  to  test  real-world  Space  Control 
operations  in  a  Y2K  environment. 
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i 

£GAO 

Accountability  *  Integrity  *  Reliability 


Background 


•  The  Space  Control  “thin  line”  included  50  systems  supporting  35  critical 
tasks.  The  OPEVAL  involved  29  of  these  systems. 

•  The  Space  Control  OPEVAL  was  completed  between  March  11-25,  1999 
and  included  these  test  execution  events: 

-  Rehearsal:  March  11,  1999. 

-  Baseline:  March  20, 1999. 

-  Surveillance/Intelligence  Testing:  March  15-19,  1999. 

-  Protection  Testing:  March  22-25,  1999. 

•  The  test  environment  consisted  of  desktop  computers;  geographically 
dispersed  ground  radar  sites;  and  partitioned  IBM  mainframes. 
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SPACECOM  Space  Control  Y2K  OPEVAL 


L 

£GAO 

Accountability  *  Integrity  *  Reliability 


•  The  required  calendar  and  leap  year  events  (September  8,  1999  to 
September  9,  1999;  December  31,  1999  to  January  1,  2000;  February 
28,  2000  to  February  29,  2000;  and  February  29,  2000  to  March  1 ,  2000) 
were  assessed  as  part  of  the  OPEVAL. 

•  The  Space  Control  OPEVAL  assessed  46  date  dependent  functions  in 
29  systems  using  these  dates. 


Background 
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i 

GAO 

. . : . .  Accountability  *  Integrity  +  Reliability 

Background 

Space  Control  Critical  Tasks 

Thin  Line  Systems 

1.  Maintain  the  current  space  environment 
database 

Navy  Fence;  Millstone;  Thule;  TOS;  Altair; 
MPDS;  IDHS;  SPADOC;  CMP;  CFE-R; 

ICIG;  NUIS;  AMHS 

2.  Space  Surveillance  Network  tasking 

MPDS;  SPADOC;  CMP 

3.  Observation  control/tasking  analysis 

MPDS;  SPADOC;  CMP 

4.  Observation  Processing 

SPADOC 

5.  Element  Set  Updates 

SPADOC 

6.  Transmit  Field  Element  Sets 

SPADOC 

7.  Cross-tag/lost  satellite/unknown  observation 
processing 

SPADOC 

8.  Launch  processing 

SPADOC 
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i 

jfe  G  A  0 

Accountability  *  Integrity  *  Reliability 


Background 


Space  Control  Critical  Tasks 

Thin  Line  Systems 

9.  Maneuver  Processing 

SPADOC 

10.  Manual  Piece  Separation  Processing 

SPADOC 

1 1 .  Collision  Avoidance 

SPADOC 

12.  Decay  Processing/re-entry  assessment 

SPADOC 

13.  Break-up  Processing 

SPADOC 

14.  Monitor  and  report  status  of  all  sensor  sites 

MPDS;  SPADOC;  CMP 

15.  Manage  sensor  coverage 

MPDS;  SPADOC;  CMP 

16.  Perform  event-related  up-channe!  reporting 

MPDS;  SPADOC;  CMP 
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i 

£  GAO 

Accountability  *  Integrity  *  Reliability 


Background 


Space  Control  Critical  Tasks 

Thin  Line  Systems 

17.  Perform  extended  collection/surveillance 
against  foreign  satellites  to  characterize  status  and 
performance  parameters  as  well  as  support  the 
foreign  space  order  of  battle 

Navy  Fence,  Millstone,  Thule,  TOS,  Altair; 
MPDS;  IDHS;  SPADOC;  CMP;  CFE-R; 

ICIG;  NUIS;  AMHS;  OSAS;  SDB;  SMAT; 
SMPAS 

18.  Monitor  the  space  situation  and  collect  and 
correlate  data  on  potential  and  actual  hostile 
activities  against  U.S.  and  designated  allied  space 
systems. 

Navy  Fence,  Millstone,  Thule,  TOS,  Altair; 
MPDS;  IDHS;  SPADOC;  CMP;  CFE-R; 

ICIG;  NUIS;  AMHS;  SDB;  SATRAN 

19.  Assess  data  and  determine  intent 

SPADOC;  AMHS;  NUIS;  OSAS;  SDB; 

SMAT;  SMPAS 

20.  Inform  the  National  Military  Command  Center  of 
impending,  current,  and  completed  hostile  space 
activity 

MPDS;  SPADOC;  CMP 

21 .  Characterize  the  results  of  a  space  attack 

SPADOC 

18 
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£GAO 

Accountabilitv  ^  lntearitv  *  Reliability 


Background 


Space  Control  Critical  Tasks 

Thin  Line  Systems 

22.  Provide  designated  authorities  with  situation 
reports 

MPDS;  SPADOC;  CMP 

23.  Provide  technical  support  as  required 

MPDS;  SPADOC;  CMP 

24.  Provide  assistance  to  routine  peacetime  space 
operations 

MPDS;  SPADOC;  CMP 

25.  Provide  warning  and  assessment  messages 

MPDS;  SPADOC;  CMP 

26.  Provide  countermeasure  coordination/status 

MPDS;  SPADOC;  CMP 

27.  Inform  space  system  owner/operators  and  other 
designated  authorities  of  selected  countermeasures 

MPDS;  SPADOC;  CMP 
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Accountability  +  Integrity  *  Reliability 


Background 


Space  Control  Critical  Tasks 

Thin  Line  Systems 

28.  Evaluate  countermeasure  effectiveness 

SPADOC 

29.  Assist  in  planning  for  further  countermeasures 

SPADOC 

30.  Inform  National  Military  Command  Center  and 
other  appropriate  elements  of  the  results  of 
countermeasure  implementation 

MPDS;  SPADOC;  CMP 

31.  Maintain  current  documentation  of  world-wide 
counter  space  capabilities 

SPADOC;  CFE-R;  ICIG;  NUIS;  SDB; 

SMPAS;  OSAS;  SMAT 

32.  Receive  taskings  (sensors) 

Navy  Fence;  Millstone;  Thule;  TOS;  Altair; 
MPDS;  CMP 

33.  Schedule  tracks  (sensors) 

Navy  Fence;  Millstone;  Thule;  TOS;  Altair 

34.  Conduct  tracks  (sensors) 

Navy  Fence;  Millstone;  Thule;  TOS;  Altair 

35.  Transmit  track  data  (sensors) 

Navy  Fence;  Millstone;  Thule;  TOS;  Altair; 
MPDS;  IDHS 
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GAO 

Accountability  *  Integrity  *  Reliability 


Results  of  GAO  Review 
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i 

M  G  A  O 

Accountability  *  integrity  *  Reliability 


Planning 


Defense  Test  Criteria 

Result 

Specify  test  assumptions  and  limitations 

Satisfied 

Establish  a  Y2K  task  force  and  assign  responsibilities 

Satisfied 

Identify  critical  missions/tasks/systems 

Satisfied 

Verify  systems  essential  to  mission  are  Y2K  compliant/certified 

Partially  Satisfied 

Develop  OPEVAL  plan  to  guide  event  planning  and  execution 

Satisfied 

Identify  and  schedule  CINC/Allied/Component/Agency  support 

Satisfied 

Determine  relevant  and  necessary  resources  (e.g.,  funding, 
personnel,  equipment,  etc.) 

Satisfied 

Ensure  approved  Y2K  contingency  plans  are  prepared 

Partially  Satisfied 

Develop  risk  management  plan 

Satisfied 

Identify  simulation  needs  and  establish  supporting  environment 

Satisfied 

Develop  data  collection  and  analysis  plan  or  approaches 

Satisfied 
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i 

GAO 

Accountability  *  integrity  +  Reliability 


Findings:  Planning 


Criteria:  In  planning  for  the  OPEVAL,  CINCs  are  to  define  assumptions 
concerning  the  readiness  of  systems  and  the  ability  to  evaluate  systems  in 
light  of  real-world  limitations. 


Finding:  Space  Command  identified  real-world  considerations  and  system 
readiness  limitations  during  Y2K  planning  meetings.  These  limitations 
were  disclosed  in  the  OPEVAL  Plan  and  to  the  JCS.  Specifically,  Space 
Command  reported  that  29  of  the  50  “thin  line”  systems  would  be  included 
and  21  would  be  excluded  from  the  OPEVAL  due  to  resource  constraints, 
or  because  they  would  be  tested  in  other  OPEVALs.  Six  of  the  21 
excluded  systems  were  communications  systems  that  were  to  be  tested  in 
a  future  OPEVAL. 
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GAO 

Accountability  *  Integrity  *  Reliability 


Findings:  Planning 


Criteria:  A  CINC  Y2K  Task  Force  composed  of  knowledgeable 
Y2K,  test,  and  systems  experts  should  be  formed  to  establish 
the  base  for  all  Y2K  planning,  coordination,  execution,  and 
reporting. 

Finding:  Consistent  with  the  defined  scope  of  the  OPEVAL, 

Space  Command  established  a  Y2K  Task  Force  and  it  defined 
roles  and  responsibilities  with  milestones  for  each  member. 
Members  of  the  task  force  included  satellite/system  specialists, 
test  and  evaluation  experts,  system  analysts,  and  public  affair 
specialists  from  the  Command’s  Operations,  Intelligence, 

Planning,  and  Public  Affairs  units.  They  also  included  Air 
Force’s  Operational  Test  and  Evaluation  Center  (AFOTEC)  and 
17th  Test  Squadron,  military  service,  and  NASA 
representatives. 
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Accountability  *  Integrity  *  Reliability 


Findings:  Planning 


Criteria:  CINCs  need  to  analyze  critical  missions  to  determine 
the  most  critical  missions  and  identify  the  critical  tasks 
supporting  each  critical  mission.  In  addition,  the  minimum 
number  of  integrated  automated  information  platforms/systems 
required  to  perform  each  critical  task  or  critical  mission  must  be 
identified  (the  “thin  line”). 

Finding:  Consistent  with  the  defined  scope  of  the  OPEVAL, 
Space  Command  identified  35  critical  tasks  that  needed  to  be 
evaluated  to  determine  mission  readiness  in  a  Y2K 
environment.  In  addition,  Space  Command  identified  a  total  of 
29  “thin  line”  systems  to  support  these  tasks. 
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Accountability  *  Integrity  *  Reliability 


F in d ings:  Planning 


Criteria:  Ensure  that  mission-critical  “thin  line”  systems  are 
certified  Y2K  compliant. 


Finding:  Consistent  with  the  defined  scope  of  the  OPEVAL, 

Space  Command  verified  that  28  of  29  mission-critical,  “thin 
line”  systems  to  be  included  in  the  OPEVAL  were  certified  Y2K 
compliant.  The  29th  system  was  not  certified  as  compliant,  but 
was  nevertheless  included  in  the  OPEVAL  rather  than  invoking 
the  system’s  contingency  plan  because,  according  to  Space 
Command  officials,  they  verified  that  the  application  on  the 
system  relevant  to  the  OPEVAL  was  compliant.  This  is  contrary 
to  JCS  guidance  and  GAO’s  end-to-end  test  guidance,  which 
defines  system  (not  application)  compliance  as  a  precondition 
to  end-to-end  testing.  In  short,  Y2K  compliance  of  an 
application  in  isolation  is  of  very  limited  value  unless  the  system 
platform  that  it  runs  on,  and  the  other  applications  running  on 

the  system  that  it  interoperates  with,  are  also  compliant. _ 
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Jpi  GAO  Finding:  Planning 

Accountability  *  Integrity  +  Reliability  w  w 


Finding :  At  the  time  of  our  October  1 , 1 999,  briefing,  Space 
Command  had  not  addressed  this  concern.  We  therefore 
recommended  that  this  deviation  be  disclosed  in  the  final 
OPEVAL  report.  Space  Command  officials  agreed  with  our 
recommendation  that  the  final  report  disclose  this  information 
and  now  plans  to  revise  the  final  report.  Additionally,  Space 
Command  stated  that  the  system  is  scheduled  to  be 
compliant  in  November  1999  and  to  be  included  in  its 
November  1999  operational  evaluation  of  the  intelligence 
mission  area. 
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GAO 

Accountability  *  Integrity  *  Reliability 


Findings:  Planning 


Criteria:  The  Y2K  task  force  should  document  how  the  OPEVAL 
will  be  conducted,  data  will  be  gathered  and  analyzed,  and  how 
reports  will  be  formatted. 

Finding:  Space  Command  developed  an  exercise  directive  and 
test  plan  for  the  OPEVAL  to: 

-  ensure  that  mechanisms  for  evaluating  critical  dates  and  contingency 
plans  for  mission-critical  systems  are  executed. 

-  document  participant  roles  and  responsibilities. 

-  link  critical  missions,  critical  tasks,  architectures,  test  cases,  and  data 
elements. 

-  report  Y2K  OPEVAL  results. 
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GAO 

Accountability  *  Integrity  +  Reliability 


Findings:  Planning 


Criteria:  When  preparing  for  a  Y2K  OPEVAL,  determine  the 
extent  of  participation  of  other  CINCs,  allies,  components,  and 
agencies  and  coordinate  their  participation  in  the  event. 


Finding:  Consistent  with  the  defined  scope  of  the  OPEVAL, 
Space  Command  identified,  coordinated,  and  scheduled  the 
OPEVAL  with  Y2K  Task  Force  members  from  the  Command’s 
Operations,  Intelligence,  Planning,  and  Public  Affairs  units. 
They  also  coordinated  activities  with  Air  Force’s  Operational 
Test  and  Evaluation  Center  (AFOTEC),  17th  Test  Squadron, 
and  Air  Force,  Army,  and  Navy  Space  Commands,  and  NASA. 
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jjfip.  GAO  Findings:  Planning 

Accountability  *  Integrity  *  Reliability 


Criteria:  The  necessary  resources  (funding,  personnel,  training, 
equipment,  time  frames,  and  external  organization  support) 
should  be  identified  and  included  in  the  plan. 

Finding :  In  November  1998,  Space  Command  issued  a  directive 
to  ensure  testing  resources  would  be  made  available  for  Y2K 
OPEVALS.  About  $8  million  was  earmarked  for  OPEVAL 
activities,  including  the  Space  Control  OPEVAL.  Space 
Command  also  coordinated  the  evaluation  scenario  and  scripts 
with  all  OPEVAL  participants,  acquired  the  systems  hardware 
and  software  to  simulate  space  control  events,  and  scheduled 
37  test  and  operator  personnel  to  help  execute  the  OPEVAL. 
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^^^^^^^Accountability  *  Integrity  *  Reliability 


Findings:  Planning 


Criteria:  To  ensure  that  Y2K  exercise  objectives  are  met,  it  is 
essential  to  have  contingency  plans  in  place  prior  to  executing 
the  OPEVAL. 

Finding  1:  Space  Command’s  approach  to  determining  whether 
contingency  plans  were  in  place  prior  to  executing  the  OPEVAL 
was  to  rely  exclusively  on  system  owners  to  ensure  that  this 
criterion  was  met.  Space  Command  did  not  take  steps  to  verify 
this  criterion.  Space  Command  has  since  initiated  a  review  of 
about  50  contingency  plans. 

Finding  2:  During  the  OPEVAL,  operators  successfully 
performed  mission  tasks  in  response  to  the  3  “hard”  system 
failures  by  invoking  workarounds. 
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Accountability  *  Integrity  *  Reliability 


Findings:  Planning 


Criteria:  CINC-unique  risk  management  plans  should  be  developed  to 
identify  and  mitigate  system-related  risks  before  they  adversely 
impact  mission  execution. 

Finding:  Space  Command  identified  OPEVAL  risks  and  strategies  for 
managing  these  risks  in  its  Space  Control  OPEVAL  Plan.  For 
example,  the  Plan  recognizes  the  risks  associated  with  confusing 
OPEVAL  sensor  observations  with  real-world  observations.  To 
mitigate  these  risks,  Space  Command’s  Plan  provides  strategies  for 
isolating  systems’  execution  of  OPEVAL  tasks/functions  from  real- 
world  system  operations  by  electronic  partitioning. 
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£  GAO 

Accountability  +  Integrity  *  Reliability 


Findings:  Planning 


Criteria:  CINCs  should  (1)  determine  if  simulations  or  manual 
data  input  will  be  needed  during  the  execution  of  the  OPEVAL, 
and,  if  needed,  (2)  ensure  that  an  environment  which  can 
support  the  simulation  is  planned  for  and  acquired. 

Finding:  Within  the  defined  scope  of  the  OPEVAL,  Space 
Command  identified  the  simulations  needed  and  manual  data 
inputs  required  for  testing  and  ensured  that  data  injection 
methodologies  were  included  in  the  OPEVAL  Plan  and  master 
scenario  events  list  (MSEL).  For  example,  Space  Command 
used  simulated  scenarios  to  perform  satellite  orbital  changes 
for  tracking  and  cataloging  purposes. 
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Accountability  *  Integrity  *  Reliability 


Findings:  Planning 


Criteria:  A  plan  should  be  prepared  to  help  coordinate  and 
synchronize  all  OPEVAL  data  collection  and  assessment 
activities. 

Finding:  Consistent  with  the  specified  scope  of  the  OPEVAL, 
Space  Command  developed  a  data  collection  and  analysis  plan 
that  included  (1)  specific  actions  that  should  be  accomplished  by 
the  OPEVAL  participants  prior  to  the  start  of  and  at  the 
completion  of  each  OPEVAL,  (2)  ground  rules  for  collecting  and 
documenting  mission-critical  system  outputs,  and  (3)  direction 
on  reviewing  the  critical  tasks  executed  during  the  OPEVAL  and 
determining  the  performance  of  the  mission-critical  “thin  line” 
systems. 
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GAO  Execution 

Accountability  *  Integrity  *  Reliability 


Defense  Test  Criteria 

Result 

Conduct  OPEVAL  rehearsal 

Satisfied 

Follow  configuration  management  policies 

Partially  Satisfied 

Perform  baseline  test  for  OPEVAL 

Satisfied 

Execute  required  Y2K  date  rollover  tests 

Satisfied 

Collect  and  archive  all  Y2K-relevant  data  and  ensure  that 
systems  are  reset  to  current  day  operations 

Satisfied 
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Accountability  *  Integrity  *  Reliability 


Findings:  Execution 


Criteria:  Prior  to  executing  the  Y2K  OPEVAL,  a  rehearsal  should  be 
conducted  to  ensure  that  all  critical  systems  and  interfaces  identified 
in  the  system  architecture  are  operating  correctly  and  that  OPEVAL 
staff  know  their  roles  and  responsibilities. 


Finding:  Space  Command  performed  a  rehearsal/test  readiness 
review  on  March  11,  1999.  The  rehearsal  was  used  to  (1)  validate 
that  the  test  readiness  review  requirements,  (2)  verify  data  collection 
and  analysis  methodologies,  (3)  confirm  the  baseline  configuration 
for  testing,  and  (4)  ensure  OPEVAL  staff  practiced  their  roles  and 
responsibilities. 
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^^^^^^^Accountability  *  Integrity  *  Reliability 


Findings:  Execution 


Criteria:  The  configurations  of  systems  and  architecture 
established  for  OPEVAL  testing  should  not  be  changed  unless 
authorized  by  the  test  director. 

t 

Finding:  Changes  were  made  to  one  system  in  the  test 
environment  after  the  baseline  for  the  OPEVAL  was  established 
and  without  authorization  from  the  test  director.  These  changes 
contributed  to  one  “hard”  system  failure  identified  during  the 
OPEVAL.  (Information  on  the  nature  of  the  system  failure  is 
classified.)  Space  Command  has  since  reinforced  the  need  to 
strictly  follow  configuration  management  policies  during 
OPEVALS  and  tasked  the  17th  Test  Squadron  and  intelligence 
unit  to  develop  ways  to  better  ensure  that  configuration 
management  over  test  baselines  is  enforced. 


37 


GAO/AIMD-00-30  Space  Command  Y2K  Operational  Testing 


Appendix  I 

Briefing  on  Results  of  GAO  Review  of 
SPACECOM  Space  Control  Y2K  OPEVAL 


&  GAO 

Accountability  *  Integrity  +  Reliability 


Findings:  Execution 


Criteria:  A  baseline  Y2K  test  should  be  executed  to  establish  expected 
results  data  that  will  be  used  to  compare  to  output  data  captured  during  the 
Y2K  date  rollover  tests  and  to  help  establish  whether  or  not  a  failure  is 
Y2K-related. 

Finding:  Space  Command  conducted  a  baseline  Y2K  test  on 
March  20,  1 999.  However,  this  test  only  covered  the  critical  tasks 
associated  with  space  surveillance  and  protection  and  did  not  include 
intelligence  tasks  because  officials  stated  that  baseline  testing  duplicated 
rehearsal  activities.  This  position  is  contrary  to  JCS  guidance.  According  to 
JCS  guidance,  the  purpose  of  the  rehearsal  is  to  provide  operators  with  an 
opportunity  to  practice  their  responsibilities.  In  contrast,  the  baseline  test  is 
to  execute  the  master  scenario  events  list  under  operational  conditions  to 
establish  expected  outputs  against  which  OPEVAL  results  can  be 
compared.  Clearly,  these  two  execution  requirements,  because  they  serve 
different  purposes,  differ  in  terms  of  content,  depth,  and  scope,  and  thus 
are  not  duplicative. 
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GAO  Findings:  Execution 

Accountability  *  Integrity  *  Reliability  w 

Finding :  Space  Command  officials  acknowledge  the 
differences  between  test  rehearsals  and  baseline  tests,  but 
explained  that  the  rehearsal  for  intelligence  systems  was 
expanded  to  satisfy  baseline  testing  requirements, 
including  using  the  same  quality  and  quantity  of  data 
planned  for  the  baseline  test.  To  verify  this,  we  reviewed 
information  subsequently  provided  by  the  test  directorate 
and  found  that  it  showed  a  level  of  testing  rigor  for  these 
systems  that  went  beyond  that  normally  required  of  a  test 
rehearsal.  We  also  verified  that  baseline  test  results  were 
documented  during  the  rehearsal.  SPACECOM  officials 
have  disclosed  this  deviation  and  its  impact  in  its  amended 
OPEVAL  report. 
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F i n d ings:  Execution 


Criteria:  Mission-critical  “thin  line”  systems  should  be  executed 
using  normal  operating  procedures,  and  a  seamless  continuity  of 
operations  during  critical  Y2K  date  rollovers  should  be  observed. 

Finding:  For  the  Space  Control  OPEVAL,  29  systems  were 
tested  and  3  experienced  “hard”  failures.  According  to  Space 
Command  officials,  it  was  not  necessary  to  invoke  contingency 
plans  for  these  failures  because  operators  were  able  to  perform 
workarounds  to  complete  mission  tasks.  For  2  of  the  3  “hard” 
failures,  these  workarounds  were  included  in  OPEVAL 
documentation;  however,  for  the  third  “hard”  failure,  OPEVAL 
documentation  was  not  prepared.  According  to  Space 
Command  officials,  in  all  3  cases,  operations  were  not  disrupted 
and  tasks  were  completed  seamlessly  and  continually. 
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Criteria:  Ensure  that  all  data  needed  to  conduct  the  evaluation 
for  the  Y2K  case  has  been  captured  prior  to  resetting  the  system 
to  current  day  operations  requirements. 

Finding:  Prior  to  resetting  the  systems  to  present  day  operational 
conditions,  Space  Command  determined  that  (1)  the  29  “thin- 
line”  systems  were  assessed,  (2)  the  master  scenario  events 
were  performed  and  deviations  were  identified,  and  (3)  all  data 
needed  to  make  an  assessment  of  Space  Command’s  ability  to 
perform  the  defined  “thin  line”  were  collected  and  archived. 
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Analysis 


Defense  Test  Criteria 

Re 

mm 

Categorize,  document,  and  report  failures 

Determine  mission  impact  of  Y2K  failures 

Ensure  Y2K  OPEVAL  exit  criteria  are  met 

Satisfied 

Satisfied 

Partially  Satisfied 
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Findings:  Analysis 


Criteria:  All  failures  are  to  be  identified  and  properly  categorized 
as  either  “hard”  or  “soft”  failures  and  should  be  documented  and 
reported  in  accordance  with  the  data  collection  and  analysis 
plan. 

Finding:  Space  Command  identified  5  Y2K  failures  during  the 
Space  Control  OPEVAL  and  categorized  3  as  “hard”  and  2  as 
“soft”  failures.  All  system  failures  were  documented  in 
accordance  with  DOD  Y2K  requirements  and  reported  to  the 
Joint  Staff  Y2K  office.  Examples  of  the  “hard”  failures  include  a 
system  that  did  not  display  messages  during  and  after  the  leap 
year  rollover  and  a  system  that  did  not  automatically  list  file 
names  for  operators  in  a  viewer  window. 
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Findings:  Analysis 


Criteria:  Determine  the  impact  of  a  failure  on  the  accomplishment 
of  a  critical  mission. 

Finding:  Space  Command  determined  that  all  5  Y2K  failures  had 
no  significant  impact  on  its  ability  to  perform  the  Space  Control 
mission. 
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Findings:  Analysis 


Criteria:  JCS  defined  9  exit  criteria  that  OPEVAL  results  should 
be  measured  to  ensure  that  critical  tasks  and  missions  can  be 
performed  in  a  Y2K  environment. 

Finding  1:  Space  Command  measured  its  OPEVAL  performance 
against  the  9  Joint  Staff  exit  criteria  and  concluded  that  the 
Space  Control  mission  can  be  successfully  performed  in  a  Y2K 
environment. 
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Findings:  Analysis 


Finding  2:  Space  Command  did  not  document  that  all  the  measures 
of  performance  established  as  exit  criteria  for  intelligence  critical 
tasks/systems  were  achieved.  Measures  of  performance  are  used 
to  determine  whether  specified  system  functions  are  performed 
within  established  time  frames.  Space  Command’s  measures  of 
performance  for  the  critical  tasks  in  its  Space  Control  OPEVAL 
included  predetermined  numbers  of  transactions  to  be  executed 
and  time  constraints  within  which  transactions  are  to  be  executed. 

However,  Space  Command  only  documented  a  portion  of  the 
predetermined  number  of  transactions  specified  in  the  OPEVAL 
Plan  for  intelligence  critical  tasks.  According  to  intelligence  officials, 
all  predefined  transactions  were  executed  successfully  but  only 
one-fifth  were  documented  because  it  was  too  time-consuming  to 
print  screens  during  testing.  In  the  absence  of  the  requisite  test 

results  documentation,  we  could  not  validate  this  claim. _ 
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Findings:  Analysis 


Finding  3:  At  the  time  of  our  briefing,  Space  Command  had  not 
addressed  this  concern.  We  therefore  recommended  that  the 
final  report  be  revised  to  reflect  this  deviation  and  to  describe  the 
actions  being  taken  to  mitigate  the  resulting  risks.  Space 
Command  officials  have  since  agreed  that  some  alternative 
measure  should  have  been  taken  to  document  all  test  results 
and  thus  ensure  the  OPEVAL’s  integrity  was  not  compromised. 
The  officials  have  also  agreed  to  our  recommendation  and  plan 
to  revise  the  final  OPEVAL  report  to  reflect  this  deviation  and  to 
ensure  that  the  November  1999  operational  evaluation  of  its 
intelligence  mission  area  provides  for  fully  documenting  test 
results. 
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Briefing  on  Results  of  GAO  Review  of 
SPACECOM  Space  Control  Y2K  OPEVAL 
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GAO  Reporting 

Accountability  *  Integrity  *  Reliability  • 


Defense  Test  Criteria 

Result 

i _  ."i 

Prepare  Y2K  reports  describing  mission  impact  and  readiness 

Provide  reports  to  Joint  Staff  J7  within  required  time  frames 

Partially  Satisfied 

Satisfied 
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Briefing  on  Results  of  GAO  Review  of 
SPACECOM  Space  Control  Y2K  OPEVAL 


Criteria:  CINCs  are  to  prepare  Y2K  reports  describing  mission 
impact  and  readiness. 

Finding  1:  Space  Command  provided  Y2K  reports  to  the  Joint 
Staff  which  concluded  that  all  critical  tasks  supporting  Space 
Control  can  be  performed  with  no  significant  impact  on 
readiness  caused  by  potential  Y2K  failures. 

Finding  2:  Space  Command  officials  stated  that  the  reports  were 
completed  as  required.  However,  the  reports  did  not  fully 
describe  the  limitations  in  the  scope  of  the  OPEVAL  and  testing 
deviations  (i.e.,  the  omission  of  6  key  communications  systems, 
the  noncompliant  system  involved  in  the  OPEVAL,  configuration 
changes  made  to  a  system  after  the  baseline  was  established 
for  the  OPEVAL,  and  the  failure  to  fully  document  that 
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GAO 

Accountability  *  Integrity  *  Reliability 


Findings:  Reporting 


established  performance  criteria  for  intelligence  tasks/systems 
were  satisfied)  or  the  extent  to  which  these  limitations  and 
deviations  affected  the  command’s  ability  to  draw  unqualified 
conclusions  about  Space  Control  mission  readiness. 

Finding  3:  Since  completing  the  Space  Control  OPEVAL,  Space 
Command  provided  documentation  that  showed  the  6 
communications  systems  were  included  in  other  OPEVAL  or 
end-to-end  tests.  Also,  Space  Command  has  recently  agreed  to 
revise  its  30-day  report  to  disclose  testing  deviations  involving 
the  use  of  a  noncompliant  system  in  the  OPEVAL,  the  failure  to 
follow  configuration  management  procedures,  and  the  failure  to 
fully  document  intelligence  tasks/systems  test  results. 
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Findings:  Reporting 


Criteria:  A  preliminary  report  is  required  within  7  calendar  days 
after  the  completion  of  the  OPEVAL  and  a  final  report  is  required 
within  30  calendar  days.  Both  reports  are  to  be  provided  to  Joint 
Staff. 

Finding:  Space  Command  completed  the  7-  and  30-day  reports 
for  the  Space  Control  OPEVAL  and  provided  them  to  Joint  Staff. 
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Briefing  on  Results  of  GAO  Review  of 
SPACECOM  Space  Control  Y2K  OPEVAL 


•  Space  Command  satisfied  many  of  the  Defense  OPEVAL 
requirements  for  its  defined  Space  Control  “thin  line.” 

•  However,  key  steps  that  are  vital  to  (1)  ensuring  that  only  compliant 
systems  or  system  contingency  plans  are  used  in  the  OPEVAL, 

(2)  fully  disclosing  deviations  from  planned  performance  measures 
and  the  impact  of  doing  so,  and  (3)  accurately  reporting  mission 
readiness  in  light  of  OPEVAL  scope  limitations  were  not  fully 
satisfied.  As  a  result,  the  Y2K  readiness  of  Space  Control  critical 
tasks  involving  intelligence  and  communications  systems  was  not 
known  with  sufficient  surety  to  support  Space  Command’s  March 
1999  unqualified  conclusion  of  mission  readiness  in  a  Y2K 
environment. 
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Conclusions 


•  Since  then,  Space  Command  has  taken  steps  to  fill  voids  in  its 
understanding  of  Space  Control  mission  readiness  by  ensuring  that 
omitted  communications  systems  were  included  in  other  Y2K  end- 
to-end  testing  or  OPEVAL  events.  It  has  also  taken  additional 
action  to  improve  testing  procedures  and  documentation 
requirements  and  has  agreed  to  address  our  recommendation  for 
revising  its  final  OPEVAL  report  to  reflect  deviations  with  regard  to 
the  performance  and  verification  of  intelligence  systems’  Y2K 
compliance.  Moreover,  Space  Command  has  acted  to  ensure  that 
its  planned  November  1999  operational  evaluation  provides  for  fully 
documenting  test  results.  Therefore,  we  are  not  making  any  further 
recommendations  at  this  time. 
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Objectives,  Scope,  and  Methodology 


At  the  request  of  the  Chairman,  House  Appropriations  Committee, 
Subcommittee  on  Defense,  we  selected  the  Space  Command  Space  Control 
evaluation  for  review  to  determine  (1)  if  the  evaluation  was  planned, 
executed,  and  documented  in  accordance  with  DOD  guidelines,  and 
(2)  what  the  evaluation  results  indicated  concerning  readiness  and  risks. 
This  operational  evaluation  was  selected  in  collaboration  with  the  Defense 
Inspector  General  to  ensure  appropriate  coverage  of  all  combatant 
command  operational  evaluations  and  no  duplication  of  effort. 

To  satisfy  our  first  objective,  we  reviewed  the  evaluation  plan,  testing 
documentation  and  records,  and  test  results  and  associated  reports.  We 
also  interviewed  Space  Command  officials  responsible  forYear  2000 
operational  evaluation  planning,  execution,  and  reporting  tasks.  Further, 
we  examined  the  century  date  rollover  testing  documents  for  the 
operational  evaluation  and  compared  Space  Command’s  operational 
evaluation  planning,  execution,  analysis,  and  reporting  actions  against  JCS 
operational  evaluation  guidance  and  our  Year  2000  testing  guide. 

To  satisfy  the  second  objective,  we  reviewed  Space  Command’s  operational 
evaluation  results,  including  its  7-  and  30-day  reports  and  system  problem 
tracking  reports.  We  also  interviewed  Space  Command  officials  and 
analysts  responsible  for  developing  operational  evaluation  assessment 
methodologies,  interpreting  evaluation  metrics,  and  ensuring  that 
evaluation  exit  criteria  were  met. 

On  October  1,  1999,  we  briefed  Space  Command  leadership  on  the  results 
of  our  review.  Space  Command  provided  oral  comments  on  our  briefing 
slides,  and  we  have  incorporated  them  as  appropriate.  We  performed  our 
work  from  March  through  October  1999  in  accordance  with  generally 
accepted  government  auditing  standards. 
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